Skip to content

Optimize think tag parser with early-exit lastIndexOf and cursor indexing - #1219

Open
nordicnode wants to merge 1 commit into
CodebuffAI:mainfrom
nordicnode:perf-streaming-think-tag-parser
Open

Optimize think tag parser with early-exit lastIndexOf and cursor indexing#1219
nordicnode wants to merge 1 commit into
CodebuffAI:mainfrom
nordicnode:perf-streaming-think-tag-parser

Conversation

@nordicnode

Copy link
Copy Markdown

Optimize think tag parser with early-exit lastIndexOf and cursor indexing

Summary

• In cli/src/utils/think-tag-parser.ts, optimize streaming think-tag detection and segment parsing.
• Previously, getPartialTagLength was called on every streamed token chunk and sequentially tested 12 hardcoded prefix strings across PARTIAL_CLOSE_PREFIXES and PARTIAL_OPEN_PREFIXES with text.endsWith(prefix). Replaced this with an $O(1)$ check using text.lastIndexOf('<'). If < does not appear within the last 7 characters of the string (since the longest partial prefix is '</think', length 7), it returns 0 immediately without evaluating prefix arrays. If < is found in the trailing window, it verifies the prefix against the constant tags.
• Measured getPartialTagLength:

  • 20,000 evaluations on 5,000-char string: 10.12 ms down to 2.91 ms (3.47x faster).
  • 100,000 realistic stream chunk batches: 84.74 ms total execution time.
    • In parseThinkTags, replaced continuous remaining = remaining.slice(...) string allocations with an index cursor let cursor = 0 to scan tags directly on the source string without allocating intermediate substrings.
    • Removed unused PARTIAL_OPEN_PREFIXES and PARTIAL_CLOSE_PREFIXES arrays.
    • Verified 100% behavioral and edge-case parity against all 22 existing unit tests.

Test plan

[✓] bun test --config=/dev/null src/utils/__tests__/think-tag-parser.test.ts — 22 pass, 0 fail
[✓] bun run --cwd cli typecheck — 0 errors
[✓] PR hygiene check passed

@codebuff-team

Copy link
Copy Markdown
Contributor

Nice work here. The core insight — deriving partial-tag detection from THINK_OPEN_TAG/THINK_CLOSE_TAG via lastIndexOf('<') instead of maintaining two hand-written prefix arrays — is a real quality win independent of the speedup: it removes a source of drift if the tag constants ever change, since previously PARTIAL_OPEN_PREFIXES/PARTIAL_CLOSE_PREFIXES had to be kept in sync by hand.

I traced through the getPartialTagLength rewrite against several edge cases (multiple < in the trailing window, strings shorter than 7 chars, suffix equal to the full tag) and it matches the original semantics in every case I checked — the lastIndexOf('<') approach is sound because a partial tag can't contain an embedded <, so the rightmost < is always the correct anchor.

The parseThinkTags cursor rewrite (replacing repeated remaining.slice(...) with indexOf(..., cursor) + index bookkeeping) is a straightforward, low-risk transformation and looks behaviorally identical to the slicing version.

Two things worth double-checking before this gets ported:

  1. Confirm THINK_OPEN_TAG/THINK_CLOSE_TAG are exported/available in this file's scope with the values I assumed ('<think>', '</think>') — the diff doesn't show their definitions.
  2. The benchmark numbers (2.91ms for 20k calls) suggest this is a very cheap hot path already; it'd help to know whether getPartialTagLength is actually called often enough in practice for this to matter, or whether the real win here is just the array removal/cleanup.

Overall this is a tight, single-purpose, tested diff with no behavior change — a reasonable candidate to port after re-verifying against the private tree's tag constants.

@codebuff-team codebuff-team added bot:triaged Classified by the community triage bot pr:port-candidate Worth porting into the private source tree labels Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:triaged Classified by the community triage bot pr:port-candidate Worth porting into the private source tree

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants